Search for: All records

Quasi-Newton methods still face significant challenges in training large-scale neural networks due to additional compute costs in the Hessian related computations and instability issues in stochastic training. A well-known method, L-BFGS that efficiently approximates the Hessian using history parameter and gradient changes, suffers convergence instability in stochastic training. So far, attempts that adapt L-BFGS to large-scale stochastic training incur considerable extra overhead, which offsets its convergence benefits in wall-clock time. In this paper, we propose mL-BFGS, a lightweight momentum-based L-BFGS algorithm that paves the way for quasi-Newton (QN) methods in large-scale distributed deep neural network (DNN) optimization. mL-BFGS introduces a nearly cost-free momentum scheme into L-BFGS update and greatly reduces stochastic noise in the Hessian, therefore stabilizing convergence during stochastic optimization. For model training at a large scale, mL-BFGS approximates a block-wise Hessian, thus enabling distributing compute and memory costs across all computing nodes. We provide a supporting convergence analysis for mL-BFGS in stochastic settings. To investigate mL-BFGS’s potential in large-scale DNN training, we train benchmark neural models using mL-BFGS and compare performance with baselines (SGD, Adam, and other quasi-Newton methods). Results show that mL-BFGS achieves both noticeable iteration-wise and wall-clock speedup. 

Microscopic robots controlled by onboard integrated circuits that walk when powered by light are realized. 

We present a platform for parallel production of standalone, untethered electronic sensors that are truly microscopic, i.e., smaller than the resolution of the naked eye. This platform heterogeneously integrates silicon electronics and inorganic microlight emitting diodes (LEDs) into a 100-μm-scale package that is powered by and communicates with light. The devices are fabricated, packaged, and released in parallel using photolithographic techniques, resulting in ∼10,000 individual sensors per square inch. To illustrate their use, we show proof-of-concept measurements recording voltage, temperature, pressure, and conductivity in a variety of environments.